{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "intro_to_neural_nets.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": [],
      "collapsed_sections": [
        "O2q5RRCKqYaU",
        "vvT2jDWjrKew",
        "copyright-notice"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "copyright-notice",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Copyright 2017 Google LLC."
      ]
    },
    {
      "metadata": {
        "id": "copyright-notice2",
        "colab_type": "code",
        "cellView": "both",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "outputs": [],
      "cell_type": "code",
      "execution_count": 0,
      "source": [
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "metadata": {
        "id": "eV16J6oUY-HN",
        "colab_type": "text",
        "slideshow": {
          "slide_type": "slide"
        }
      },
      "source": [
        " # Introducci\u00f3n a las redes neuronales"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "_wIcUFLSKNdx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " **Objetivos de aprendizaje:**\n",
        "  * definir una red neuronal (RN) y sus capas ocultas a trav\u00e9s de la clase `DNNRegressor` de TensorFlow\n",
        "  * entrenar una red neuronal para aprender no linealidades en un conjunto de datos y lograr un mejor rendimiento que un modelo de regresi\u00f3n lineal"
      ]
    },
    {
      "metadata": {
        "id": "_ZZ7f7prKNdy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " En los ejercicios anteriores, usamos atributos sint\u00e9ticos para ayudar a nuestro modelo a incorporar no linealidades.\n",
        "\n",
        "Hab\u00eda un conjunto de no linealidades importante en torno a latitud y longitud, pero pueden existir otros.\n",
        "\n",
        "Por el momento tambi\u00e9n volveremos a una tarea de regresi\u00f3n est\u00e1ndar, en lugar de la tarea de regresi\u00f3n log\u00edstica del ejercicio anterior. Esto significa que prediremos `median_house_value` directamente."
      ]
    },
    {
      "metadata": {
        "id": "J2kqX6VZTHUy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## Preparaci\u00f3n\n",
        "\n",
        "Primero, carguemos y preparemos los datos."
      ]
    },
    {
      "metadata": {
        "id": "AGOM1TUiKNdz",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "from __future__ import print_function\n",
        "\n",
        "import math\n",
        "\n",
        "from IPython import display\n",
        "from matplotlib import cm\n",
        "from matplotlib import gridspec\n",
        "from matplotlib import pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn import metrics\n",
        "import tensorflow as tf\n",
        "from tensorflow.python.data import Dataset\n",
        "\n",
        "tf.logging.set_verbosity(tf.logging.ERROR)\n",
        "pd.options.display.max_rows = 10\n",
        "pd.options.display.float_format = '{:.1f}'.format\n",
        "\n",
        "california_housing_dataframe = pd.read_csv(\"https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv\", sep=\",\")\n",
        "\n",
        "california_housing_dataframe = california_housing_dataframe.reindex(\n",
        "    np.random.permutation(california_housing_dataframe.index))"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2I8E2qhyKNd4",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def preprocess_features(california_housing_dataframe):\n",
        "  \"\"\"Prepares input features from California housing data set.\n",
        "\n",
        "  Args:\n",
        "    california_housing_dataframe: A Pandas DataFrame expected to contain data\n",
        "      from the California housing data set.\n",
        "  Returns:\n",
        "    A DataFrame that contains the features to be used for the model, including\n",
        "    synthetic features.\n",
        "  \"\"\"\n",
        "  selected_features = california_housing_dataframe[\n",
        "    [\"latitude\",\n",
        "     \"longitude\",\n",
        "     \"housing_median_age\",\n",
        "     \"total_rooms\",\n",
        "     \"total_bedrooms\",\n",
        "     \"population\",\n",
        "     \"households\",\n",
        "     \"median_income\"]]\n",
        "  processed_features = selected_features.copy()\n",
        "  # Create a synthetic feature.\n",
        "  processed_features[\"rooms_per_person\"] = (\n",
        "    california_housing_dataframe[\"total_rooms\"] /\n",
        "    california_housing_dataframe[\"population\"])\n",
        "  return processed_features\n",
        "\n",
        "def preprocess_targets(california_housing_dataframe):\n",
        "  \"\"\"Prepares target features (i.e., labels) from California housing data set.\n",
        "\n",
        "  Args:\n",
        "    california_housing_dataframe: A Pandas DataFrame expected to contain data\n",
        "      from the California housing data set.\n",
        "  Returns:\n",
        "    A DataFrame that contains the target feature.\n",
        "  \"\"\"\n",
        "  output_targets = pd.DataFrame()\n",
        "  # Scale the target to be in units of thousands of dollars.\n",
        "  output_targets[\"median_house_value\"] = (\n",
        "    california_housing_dataframe[\"median_house_value\"] / 1000.0)\n",
        "  return output_targets"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pQzcj2B1T5dA",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "# Choose the first 12000 (out of 17000) examples for training.\n",
        "training_examples = preprocess_features(california_housing_dataframe.head(12000))\n",
        "training_targets = preprocess_targets(california_housing_dataframe.head(12000))\n",
        "\n",
        "# Choose the last 5000 (out of 17000) examples for validation.\n",
        "validation_examples = preprocess_features(california_housing_dataframe.tail(5000))\n",
        "validation_targets = preprocess_targets(california_housing_dataframe.tail(5000))\n",
        "\n",
        "# Double-check that we've done the right thing.\n",
        "print(\"Training examples summary:\")\n",
        "display.display(training_examples.describe())\n",
        "print(\"Validation examples summary:\")\n",
        "display.display(validation_examples.describe())\n",
        "\n",
        "print(\"Training targets summary:\")\n",
        "display.display(training_targets.describe())\n",
        "print(\"Validation targets summary:\")\n",
        "display.display(validation_targets.describe())"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RWq0xecNKNeG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## Creaci\u00f3n de una red neuronal\n",
        "\n",
        "La RN se define a trav\u00e9s de la clase [DNNRegressor](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNRegressor).\n",
        "\n",
        "Usa **`hidden_units`** para definir la estructura de la RN. El argumento `hidden_units` proporciona una lista de enteros, en la que cada entero corresponde a una capa oculta e indica el n\u00famero de nodos en ella. Por ejemplo, considera la siguiente asignaci\u00f3n:\n",
        "\n",
        "`hidden_units=[3,10]`\n",
        "\n",
        "La asignaci\u00f3n anterior especifica una red neuronal con dos capas ocultas:\n",
        "\n",
        "* La primera capa oculta contiene 3 nodos.\n",
        "* La segunda capa oculta contiene 10 nodos.\n",
        "\n",
        "Si quisi\u00e9ramos agregar m\u00e1s capas, incorporar\u00edamos m\u00e1s enteros a la lista. Por ejemplo, `hidden_units=[10,20,30,40]` crear\u00eda cuatro capas con diez, veinte, treinta y cuarenta unidades, respectivamente.\n",
        "\n",
        "De forma predeterminada, todas las capas ocultas usar\u00e1n activaci\u00f3n ReLu y estar\u00e1n totalmente conectadas."
      ]
    },
    {
      "metadata": {
        "id": "ni0S6zHcTb04",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def construct_feature_columns(input_features):\n",
        "  \"\"\"Construct the TensorFlow Feature Columns.\n",
        "\n",
        "  Args:\n",
        "    input_features: The names of the numerical input features to use.\n",
        "  Returns:\n",
        "    A set of feature columns\n",
        "  \"\"\" \n",
        "  return set([tf.feature_column.numeric_column(my_feature)\n",
        "              for my_feature in input_features])"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "zvCqgNdzpaFg",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def my_input_fn(features, targets, batch_size=1, shuffle=True, num_epochs=None):\n",
        "    \"\"\"Trains a neural net regression model.\n",
        "  \n",
        "    Args:\n",
        "      features: pandas DataFrame of features\n",
        "      targets: pandas DataFrame of targets\n",
        "      batch_size: Size of batches to be passed to the model\n",
        "      shuffle: True or False. Whether to shuffle the data.\n",
        "      num_epochs: Number of epochs for which data should be repeated. None = repeat indefinitely\n",
        "    Returns:\n",
        "      Tuple of (features, labels) for next data batch\n",
        "    \"\"\"\n",
        "    \n",
        "    # Convert pandas data into a dict of np arrays.\n",
        "    features = {key:np.array(value) for key,value in dict(features).items()}                                             \n",
        " \n",
        "    # Construct a dataset, and configure batching/repeating.\n",
        "    ds = Dataset.from_tensor_slices((features,targets)) # warning: 2GB limit\n",
        "    ds = ds.batch(batch_size).repeat(num_epochs)\n",
        "    \n",
        "    # Shuffle the data, if specified.\n",
        "    if shuffle:\n",
        "      ds = ds.shuffle(10000)\n",
        "    \n",
        "    # Return the next batch of data.\n",
        "    features, labels = ds.make_one_shot_iterator().get_next()\n",
        "    return features, labels"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "U52Ychv9KNeH",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def train_nn_regression_model(\n",
        "    learning_rate,\n",
        "    steps,\n",
        "    batch_size,\n",
        "    hidden_units,\n",
        "    training_examples,\n",
        "    training_targets,\n",
        "    validation_examples,\n",
        "    validation_targets):\n",
        "  \"\"\"Trains a neural network regression model.\n",
        "  \n",
        "  In addition to training, this function also prints training progress information,\n",
        "  as well as a plot of the training and validation loss over time.\n",
        "  \n",
        "  Args:\n",
        "    learning_rate: A `float`, the learning rate.\n",
        "    steps: A non-zero `int`, the total number of training steps. A training step\n",
        "      consists of a forward and backward pass using a single batch.\n",
        "    batch_size: A non-zero `int`, the batch size.\n",
        "    hidden_units: A `list` of int values, specifying the number of neurons in each layer.\n",
        "    training_examples: A `DataFrame` containing one or more columns from\n",
        "      `california_housing_dataframe` to use as input features for training.\n",
        "    training_targets: A `DataFrame` containing exactly one column from\n",
        "      `california_housing_dataframe` to use as target for training.\n",
        "    validation_examples: A `DataFrame` containing one or more columns from\n",
        "      `california_housing_dataframe` to use as input features for validation.\n",
        "    validation_targets: A `DataFrame` containing exactly one column from\n",
        "      `california_housing_dataframe` to use as target for validation.\n",
        "      \n",
        "  Returns:\n",
        "    A `DNNRegressor` object trained on the training data.\n",
        "  \"\"\"\n",
        "\n",
        "  periods = 10\n",
        "  steps_per_period = steps / periods\n",
        "  \n",
        "  # Create a DNNRegressor object.\n",
        "  my_optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
        "  my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
        "  dnn_regressor = tf.estimator.DNNRegressor(\n",
        "      feature_columns=construct_feature_columns(training_examples),\n",
        "      hidden_units=hidden_units,\n",
	"      optimizer=my_optimizer\n",
        "  )\n",
        "  \n",
        "  # Create input functions.\n",
        "  training_input_fn = lambda: my_input_fn(training_examples, \n",
        "                                          training_targets[\"median_house_value\"], \n",
        "                                          batch_size=batch_size)\n",
        "  predict_training_input_fn = lambda: my_input_fn(training_examples, \n",
        "                                                  training_targets[\"median_house_value\"], \n",
        "                                                  num_epochs=1, \n",
        "                                                  shuffle=False)\n",
        "  predict_validation_input_fn = lambda: my_input_fn(validation_examples, \n",
        "                                                    validation_targets[\"median_house_value\"], \n",
        "                                                    num_epochs=1, \n",
        "                                                    shuffle=False)\n",
        "\n",
        "  # Train the model, but do so inside a loop so that we can periodically assess\n",
        "  # loss metrics.\n",
        "  print(\"Training model...\")\n",
        "  print(\"RMSE (on training data):\")\n",
        "  training_rmse = []\n",
        "  validation_rmse = []\n",
        "  for period in range (0, periods):\n",
        "    # Train the model, starting from the prior state.\n",
        "    dnn_regressor.train(\n",
        "        input_fn=training_input_fn,\n",
        "        steps=steps_per_period\n",
        "    )\n",
        "    # Take a break and compute predictions.\n",
        "    training_predictions = dnn_regressor.predict(input_fn=predict_training_input_fn)\n",
        "    training_predictions = np.array([item['predictions'][0] for item in training_predictions])\n",
        "    \n",
        "    validation_predictions = dnn_regressor.predict(input_fn=predict_validation_input_fn)\n",
        "    validation_predictions = np.array([item['predictions'][0] for item in validation_predictions])\n",
        "    \n",
        "    # Compute training and validation loss.\n",
        "    training_root_mean_squared_error = math.sqrt(\n",
        "        metrics.mean_squared_error(training_predictions, training_targets))\n",
        "    validation_root_mean_squared_error = math.sqrt(\n",
        "        metrics.mean_squared_error(validation_predictions, validation_targets))\n",
        "    # Occasionally print the current loss.\n",
        "    print(\"  period %02d : %0.2f\" % (period, training_root_mean_squared_error))\n",
        "    # Add the loss metrics from this period to our list.\n",
        "    training_rmse.append(training_root_mean_squared_error)\n",
        "    validation_rmse.append(validation_root_mean_squared_error)\n",
        "  print(\"Model training finished.\")\n",
        "\n",
        "  # Output a graph of loss metrics over periods.\n",
        "  plt.ylabel(\"RMSE\")\n",
        "  plt.xlabel(\"Periods\")\n",
        "  plt.title(\"Root Mean Squared Error vs. Periods\")\n",
        "  plt.tight_layout()\n",
        "  plt.plot(training_rmse, label=\"training\")\n",
        "  plt.plot(validation_rmse, label=\"validation\")\n",
        "  plt.legend()\n",
        "\n",
        "  print(\"Final RMSE (on training data):   %0.2f\" % training_root_mean_squared_error)\n",
        "  print(\"Final RMSE (on validation data): %0.2f\" % validation_root_mean_squared_error)\n",
        "\n",
        "  return dnn_regressor"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2QhdcCy-Y8QR",
        "colab_type": "text",
        "slideshow": {
          "slide_type": "slide"
        }
      },
      "cell_type": "markdown",
      "source": [
        " ## Tarea\u00a01: Entrenar un modelo de RN\n",
        "**Ajusta los hiperpar\u00e1metros con la intenci\u00f3n de disminuir el RMSE por debajo de 110.**\n",
        "\nEjecuta el siguiente bloque para entrenar un modelo de RN.  \n\n",
        "Recuerda que, en el ejercicio de regresi\u00f3n lineal, con muchos atributos, un RMSE de aproximadamente 110 era bastante bueno. Intentaremos mejorarlo.\n",
        "Tu tarea aqu\u00ed es modificar las distintas configuraciones de aprendizaje para mejorar la exactitud en los datos de validaci\u00f3n.\n\n",
        "El sobreajuste es un verdadero riesgo potencial para las RN. Puedes observar la brecha entre la p\u00e9rdida en los datos de entrenamiento y la p\u00e9rdida en los datos de validaci\u00f3n para determinar si tu modelo est\u00e1 comenzando a tener un sobreajuste. Si la brecha comienza a crecer, por lo general es un indicador seguro de sobreajuste.\n\n",
        "Debido al n\u00famero de las diferentes configuraciones posibles, se recomienda enf\u00e1ticamente tomar nota de cada prueba como gu\u00eda para el proceso de desarrollo.\n\n",
        "Adem\u00e1s, cuando obtengas una buena configuraci\u00f3n, prueba ejecutarla varias veces y observa qu\u00e9 tan constante es el resultado. Las ponderaciones de RN suelen inicializarse con valores aleatorios peque\u00f1os, de manera que debes observar las diferencias entre una ejecuci\u00f3n y otra.\n\n"
      ]
    },
    {
      "metadata": {
        "id": "rXmtSW1yKNeK",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "dnn_regressor = train_nn_regression_model(\n",
        "    learning_rate=0.01,\n",
        "    steps=500,\n",
        "    batch_size=10,\n",
        "    hidden_units=[10, 2],\n",
        "    training_examples=training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "O2q5RRCKqYaU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ### Soluci\u00f3n\n",
        "\n",
        "Haz clic m\u00e1s abajo para ver una soluci\u00f3n posible."
      ]
    },
    {
      "metadata": {
        "id": "j2Yd5VfrqcC3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " **NOTA:** Esta selecci\u00f3n de par\u00e1metros es algo arbitraria. Aqu\u00ed probamos combinaciones que son cada vez m\u00e1s complejas, combinadas con el entrenamiento durante m\u00e1s tiempo, hasta que el error se encuentra por debajo de nuestro objetivo. Esta no es de ning\u00fan modo la mejor combinaci\u00f3n; otras pueden alcanzar un RMSE incluso m\u00e1s bajo. Si intentas encontrar el modelo que pueda alcanzar el mejor error, deber\u00e1s usar un proceso m\u00e1s riguroso, como una b\u00fasqueda de par\u00e1metros."
      ]
    },
    {
      "metadata": {
        "id": "IjkpSqmxqnSM",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "dnn_regressor = train_nn_regression_model(\n",
        "    learning_rate=0.001,\n",
        "    steps=2000,\n",
        "    batch_size=100,\n",
        "    hidden_units=[10, 10],\n",
        "    training_examples=training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "c6diezCSeH4Y",
        "colab_type": "text",
        "slideshow": {
          "slide_type": "slide"
        }
      },
      "cell_type": "markdown",
      "source": [
        " ## Tarea\u00a02: Evaluar los datos de prueba\n",
        "\n",
        "**Confirma que los resultados de tu rendimiento de validaci\u00f3n respalden los datos de prueba.**\n",
        "\n",
        "Una vez que tengas un modelo con el que est\u00e9s satisfecho, eval\u00faalo con los datos de prueba para compararlos con el rendimiento de validaci\u00f3n.\n",
        "\n",
        "Recuerda que el conjunto de datos de prueba est\u00e1 ubicado [aqu\u00ed](https://download.mlcc.google.com/mledu-datasets/california_housing_test.csv)."
      ]
    },
    {
      "metadata": {
        "id": "icEJIl5Vp51r",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "test": {
            "output": "ignore",
            "timeout": 600
          }
        },
        "cellView": "both"
      },
      "source": [
        "california_housing_test_data = pd.read_csv(\"https://download.mlcc.google.com/mledu-datasets/california_housing_test.csv\", sep=\",\")\n",
        "\n",
        "# YOUR CODE HERE"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vvT2jDWjrKew",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ### Soluci\u00f3n\n",
        "\n",
        "Haz clic m\u00e1s abajo para ver una soluci\u00f3n posible."
      ]
    },
    {
      "metadata": {
        "id": "FyDh7Qy6rQb0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " De manera similar a lo que hace el c\u00f3digo de la parte superior, tenemos que cargar el archivo de datos adecuado, procesarlo previamente y llamar a predict y mean_squared_error.\n",
        "\n",
        "Ten en cuenta que no necesitamos aleatorizar los datos de prueba, dado que usaremos todos los registros."
      ]
    },
    {
      "metadata": {
        "id": "vhb0CtdvrWZx",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "california_housing_test_data = pd.read_csv(\"https://download.mlcc.google.com/mledu-datasets/california_housing_test.csv\", sep=\",\")\n",
        "\n",
        "test_examples = preprocess_features(california_housing_test_data)\n",
        "test_targets = preprocess_targets(california_housing_test_data)\n",
        "\n",
        "predict_testing_input_fn = lambda: my_input_fn(test_examples, \n",
        "                                               test_targets[\"median_house_value\"], \n",
        "                                               num_epochs=1, \n",
        "                                               shuffle=False)\n",
        "\n",
        "test_predictions = dnn_regressor.predict(input_fn=predict_testing_input_fn)\n",
        "test_predictions = np.array([item['predictions'][0] for item in test_predictions])\n",
        "\n",
        "root_mean_squared_error = math.sqrt(\n",
        "    metrics.mean_squared_error(test_predictions, test_targets))\n",
        "\n",
        "print(\"Final RMSE (on test data): %0.2f\" % root_mean_squared_error)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    }
  ]
}
